¶ … Program Evaluation to Health Care Managers Program evaluation is an important part of the health program planning, implementation, review, and change process. Patton (1997) defines program evaluation as "the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming." As this definition suggests, program evaluations can be carried out to serve different purposes, for example:

To identify and describe potential program clients and their service needs.

To improve program operations.

To determine whether the program has achieved its objectives.

To assess the program's impact.

There are a variety of different approaches and benefits to evaluation (Porteous, et al., 1997). Therefore, in order to maximize the potential usefulness of an evaluation, managers must choose the approach or approaches that provide the highest quality, most useful information to answer evaluation questions given the program context and the resources available to conduct the investigation.

This paper addresses the key issues regarding program evaluation, in an effort to determine its value to healthcare management. It will also demonstrate how and why different approaches to program evaluation should be considered and how they can be most useful to managers.

The Basics of Program Evaluation

When examining program evaluation, it is important to look at the term "program" in general. In most cases, organizations base their work on their mission to identify several overall goals that must be reached to accomplish their mission (McNamara, 1998).

In the health care industry, each of these goals is often a program. Health care programs are organized methods to provide certain related services to patients. Therefore, programs must be evaluated to decide if the programs are useful to clients. For a pharmaceutical company, for example, a program is often a one-time effort to produce a new product.

In this light, program evaluation means thoroughly and carefully collecting data about a program in an effort to make necessary decisions about the program. According to McNamara (1998), "Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments, accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc."

The type of evaluation taken improve programs depends on what managers want to learn about the program. Therefore, managers must concentrate on what information they need to make the necessary program decisions, as well as how they can accurately collect and understand that information.

Basic Myths About Program Evaluation

Some managers believe that program evaluation holds little value, arguing that it often produces hoards of banal information with useless conclusions (McNamara, 1998). In the past, this was a major problem, as program evaluation methods were selected mainly on the basis of complete scientific accuracy, reliability and validity. This approach resulted in extensive data from which only scientifically accurate conclusions were drawn. For the most part, generalizations and recommendations were avoided in favor of facts.

As a result, evaluation reports basically just stated the obvious and left program administrators critical of the value of evaluation. However, today, evaluation focuses much more on utility, relevance and practicality, rather than just concentrating on scientific validity.

Another myth regarding program evaluation is that it is specifically designed to determine the success or failure of a program. This myth assumes that success is implementing the perfect program and never having to deal with employees, customers or clients again. However, this is not a realistic goal. Instead, success is remaining open to continuous feedback and adjusting the program to meet changing needs. Evaluation provides manager with this continuing feedback.

In addition, many people believe that evaluation is a unique and complex process that occurs at a specific time in a specific way, and almost always includes the participation of outside experts. Many managers feel that they must completely understand terms like validity and reliability. However, this is not true.

Managers must consider what information they need in order to make current decisions about program issues or needs (McNamara, 1998). In addition, they have to be willing to commit to understanding what is really going on.

Many managers regularly undertake some nature of program evaluation and fail. This is because they do not take the time to understand and do not perform it in a formal manner. Thus, they fail to draw accurate conclusions and basically waste their time. Many researchers argue that program evaluation cannot be accurate if not done methodically. As a result, these managers miss significant opportunities to make more of difference for their customer and clients,...

While most have some basic measures of the outcomes, cost, and quality of an organization's services, many have questions about aspects of the cost-effectiveness of the programs and their real value to clients.
For many health care organizations, managers need to determine exactly what they need to invest in order to obtain better performance measures for their managed care programs. There are several factors that contribute to designing this type of evaluation: programs, customers, and the organization's willingness to use research results. Understanding the answers to the following questions can be extremely useful for a manager in determining whether an investment in program evaluation is worthwhile for the organization (Grannemann, 2002).

1. What is program evaluation? Program evaluation is a scientific approach used to determine the value of managed care programs. The methods of formal program evaluation have been traditionally used to assess health care programs like Medicare and Medicaid. Program evaluation helps manager to assess costs and outcomes. It usually includes a scientific research design and statistical analysis designed to measure program effects. Analysis may control for the following factors: case mix, characteristics of the local health care system, and price differences. This process may include cost-benefit or cost-effectiveness analysis.

Due to the fact that program evaluation focuses on measuring program effects, it goes beyond simple outcomes measurement or benchmarking. It also provides valuable feedback about the effects a program achieves, how it achieves them, and what its value is to the managed care organization and its clients.

2. Will it help an organization gain a competitive advantage? Program evaluation can help a health care organization gain a competitive edge by helping managers make better strategic decisions and helping managers make ongoing improvements in quality and efficiency.

Basically, managers need accurate information when faced with strategic decisions, such as what services to offer, how to introduce new products, what type of new products to make, or whether to adopt a new health care management technique. Program evaluation provides this information and enables managers to identify what contributions a program will make to the organization. It helps managers see what program features will produce the desired results and should be replicated in future programs.

Many managers believe that the best benefit of program evaluation is the way it helps move an organization more quickly in terms of developing more efficient and cost-effective programs. Program evaluation shows managers what works before their competitors do. Evaluation can help managers update an organization's managed health care programs to beat the competition. Better programs enable organizations to offer their clients lower costs or superior services.

3. Will it help with marketing? Basically, program evaluations can help managers keep their clients more informed about their programs. The stronger the clients' interest in quality and value, the greater is the importance of documenting the effectiveness of programs. For example, in many markets, employers or health care purchasing coalitions examine evidence of quality and cost-effectiveness in their local market. If the employers in a particular market use this type of information in selecting their managed care providers, organizations can gain a competitive edge if they provide credible research-based evidence of the cost-effectiveness of health care programs.

In addition, by showing the benefits of particular programs, evaluation enables managers to see how to position various programs in the market to take advantage of real strengths and ability to produce results. Evaluation results also help managers make that position clear to their clients.

4. Is evaluation worth the cost? Each evaluation project should be weighed by its net benefits to the organization. The costs can be more easily weighed than the benefits. Costs of research design, data collection, and analysis are easily estimated once an evaluation project is designed. Benefits, in terms of lower costs and/or improved outcomes, accumulate to the managed care organization, its business customers, and patients.

In many cases, it is difficult to determine how much of this benefit is due to the program evaluation project. Evaluation is more of an enabling tool that helps managers develop cost-effective solutions and make the best strategic decisions. It also helps employers to see the value of a managed care program and allows patients to receive better care that will restore good health.

As an enabling tool for decision makers, program evaluation gets its value from the importance of the program that is being evaluated. Thus, evaluation is more important when applied to large programs and when it will help managers make important…